\pagestyle{myheadings} \setcounter{page}{1} \setcounter{footnote}{0}

\section{~Mosaic approach with non-regular grids} \label{app:scrip}
\newcounters 

\vssub
\subsection{~Introduction} \label{sec:scripA}
\vssub

\noindent
\ws\ version 3.14 \citep{tol:MMAB09a} introduced multi-grid capability. This
capability is described above (\para\ref{sub:two_way}). With model version 4,
there is an option to use irregular grids or unstructured grids, as described
in \para\ref{sub:num_space_curv} and \para\ref{sub:num_space_tri},
respectively. Unfortunately, the methods described in \para\ref{sub:two_way}
are not general, as they are intended for regular grids only. Some new
capabilities are implemented in \WWver\ to accommodate irregular and unstructured
grids within the multi-grid approach.

The core component for communication from lower rank grids to higher rank
grids of \cite{tol:OMOD08b} is an interpolation in space to provide boundary
data at the higher spatial resolution. For version \WWver\, the technique was
generalized by making calls to the grid-search-utility (GSU) implemented in
\ws\ version 4 by T. Campbell. Other generalizations were made to ancillary
components of this routine.

The core component for communication from higher rank grids to lower rank
grids of \cite{tol:OMOD08b} is a conservative remapping operation: the
spectral density of a larger (low rank) grid cell is updated based on the
spectral densities of the overlapping smaller (high rank) grid cells, weighted
according to the fraction of the larger cell that is covered by each smaller
cell, keeping in mind that a smaller cell may be overlapping with more than
one larger cell. For version \WWver\, the technique was generalized by making calls to
an external software package, SCRIP-WW3, which is described below. The
remapping weights are stored in a FORTRAN ``derived data type''
array. Generalizations were also made to ancillary components of the remapping
routine, for example to the logic used to calculate distances to the
boundaries, to deal with masked points and land points, etc.

\vssub
\subsection{~SCRIP-WW3} \label{sec:scripB}
\vssub

The SCRIP-WW3 software package is adapted from the {\F SCRIP} (Spherical
Coordinate Remapping and Interpolation Package) software package of
\cite{man:Jones98}, which we refer to here as SCRIP-LANL. SCRIP-WW3 is based
on SCRIP-LANL v1.5. The primary difference between SCRIP-LANL and SCRIP-WW3 is
that the former is a standalone code using NetCDF files for user interface,
and the latter is modified to run within \ws\, with communication via system
memory. Further, SCRIP-WW3 only utilizes the conservative remapping feature,
whereas SCRIP-LANL has a number of other optional uses, such as bi-linear
remapping.

The conservative remapping used in SCRIP is based on \cite{art:Jones99}. In
this method, for each source/destination grid pair, line integrals are
computed around all cells in each grid while keeping track of intersections
with the other grid, resulting in area of overlap between grids. The method is
designed for use with a spherical coordinate system (as opposed to treating
latitudes and longitudes as if they are x- any y-axes in a Cartesian system)
and includes special logic for handling longitude wrapping (the so-called
``branch cut'') and cells that include a pole. It also allows for unstructured
grids, with arbitrary number of cell corners. The grid corner coordinates must
be given in an order which traces the outside of a grid cell in a
counterclockwise direction. The software allows either first- or second-order
remapping; weights for both are calculated in SCRIP-WW3. At present, only the
first-order remapping is implemented in \ws\ : \cite{art:Jones99} points out
that there is virtually no advantage to using the second-order method when
mapping from a fine grid to a coarse grid.

\vssub
\subsection{~SCRIP Operation} \label{sec:scripC}
\vssub

SCRIP-WW3 is activated by including {\F SCRIP} in the file {\file switch}. If
the user attempts to use irregular or unstructured grids within {\file
  ww3\_multi} without this switch, this will result in an error message and
program termination. SCRIP-WW3 is not required for {\file ww3\_shel}
(traditional one-way nesting), and is not required for {\file ww3\_multi} with
only regular grids, since original methods for remapping are retained in the
code for this purpose. SCRIP-WW3 source files are kept in a separate
directory {\dir /ftn/SCRIP/}, since it is modified 3rd party software. With
the {\F SCRIP} switch, the build system ( {\file ww3\_make} ) will
automatically compile files from this directory and link them into {\file
  ww3\_multi}.

A user may also optionally include the switch {\F SCRIPNC} along with {\F
  SCRIP}. This feature requires NetCDF. Instructions for using NetCDF in \ws\
are found in \para\ref{sec:comp} and in the file {\file w3\_make}. With {\F
  SCRIPNC} activated, for each source/destination grid pair, a NetCDF file
will be created, e.g. {\file rmp\_src\_to\_dst\_conserv\_002\_001.nc}, with
{\file 002} and {\file 001} referring to the source and destination grid
respectively; the numbering of grids is assigned by {\file ww3\_multi} and is
indicated in screen output of that program. This {\file .nc} file contains all
information required by \ws\ for remapping. Additional diagnostic information
about the remapping can be included in the {\file .nc} file by adding the
switch {\F T38}. Note: {\file switch} should include either {\F `SCRIP
  SCRIPNC'} or {\F `SCRIP'}; using {\F SCRIPNC} without {\F SCRIP} will result
in a compile error.

Though it is not required, SCRIP-WW3 may be utilized for remapping between
regular grids. In the case of spherical (lat/lon) grids, there may be slight
differences using SCRIP-WW3, since SCRIP-WW3 calculates areas based on real
distances, and the non-SCRIP approach uses degrees lat/lon.

\vssub
\subsection{~Optimization and common problems} \label{sec:scripD}
\vssub

SCRIP-WW3 routines are not parallelized. Therefore, if {\file ww3\_multi} is
run with many processes, each process will perform identical calculations of
all weights. For remapping between grids with large numbers of points, this
can make the preparations for {\file ww3\_multi} time-consuming, e.g. 3 to 10
minutes, which can be prohibitively expensive for routine, operational use. To
deal with this problem, SCRIP-WW3 has been adapted to allow use of remapping
weights that were computed in a prior application of {\file ww3\_multi}. If
the appropriate {\file .nc} files are found by {\file ww3\_multi}, it will
simply read the remapping data from these files, and SCRIP will not be
called. Of course, if any grids have been changed since the prior run, or if
moving grids are used, pre-computed weights should not be used.

An additional feature is provided for user convenience: if a file named {\file
  SCRIP\_STOP} is found in the run directory, {\file ww3\_multi} will
terminate after the {\file .nc} files are created. The content of {\file
  SCRIP\_STOP} is unimportant; it may be an empty file. When this feature is
used, remapping operations will be distributed among processes: {\file
  rmp\_src\_to\_dst\_conserv\_002\_001.nc} is created by process 1, {\file
  rmp\_src\_to\_dst\_conserv\_003\_001.nc} is created by process 2, etc.,
which will dramatically improve performance in cases where a significant
number of grids are used. To clarify, there are two modes of operation that are
 targetted with this feature: Mode A) Precalculate weights, where 
{\file SCRIP\_STOP} exists and  {\file .nc} files do not exist. Mode B) Use 
precalculated weights, where {\file SCRIP\_STOP} does not exist and {\file .nc}
 files do exist. If both files types exist (through accident) in the work 
directory, {\file ww3\_multi} will fail with an error. In a hypothetical 
operational context, Mode A is used for the first run and Mode B is used for 
all subsequent runs with the same grid set. The scalability is limited by 
the most expensive remapping pair, i.e. load balancing is an issue. 
For a case where 12 remapping pairs are calculated and each pair requires 
1/12th of the computation time, speed-up will be by factor twelve. For 
another case with 12 remapping pairs, where one remapping pair takes 50\%  
of computation time, speed-up will be by factor two only. Note that resources
are maximized by using a number of processes equal to the number of remapping
pairs: extra processes will not be used. 

To further explain the options available to users, take an example of a 
multi-grid system with 9 grids and 12 remapping pairs, with many sea points, 
run twice a day for several months, for a total of 1000 forecasts. The user 
may handle this in different ways: 

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 15mm \labelwidth 7mm
             \rightmargin 5mm \itemsep 0mm \parsep 0mm}

\item  Using {\F SCRIP}, {\F SCRIPNC}, and {\F MPI}, and using the {\file 
SCRIP\_STOP} feature, the calculation of weights will be done in parallel. 
The first time the model is applied, this may take 5 to 10 minutes to 
calculate remapping weights (Mode A above) and 20 minutes to perform the 
model forecast (Mode B above). For forecasts 2 to 1000, only the 20 minutes 
to perform the model forecast (Mode B) is needed. 

\item Using {\F SCRIP}, {\F SCRIPNC} and the {\file SCRIP\_STOP} feature, 
but creating the {\file .nc} files running in serial mode and running the 
forecast with {\F MPI}, the first time the model is applied, this may take 
20 minutes to calculate remapping weights and 20 minutes to perform the 
model forecast. For forecasts 2 to 1000, only the 20 minutes to perform 
the model forecast is needed. 

\item  Using {\F SCRIP}, {\F SCRIPNC}, and {\F MPI}, without using the 
{\file SCRIP\_STOP} feature, the calculation of weights will not be done 
in parallel, and will even be slower than if run in serial, because of 
communications. The first time the model is applied, this may take 40 
minutes to calculate remapping weights and 20 minutes to perform the model 
forecast.  For forecasts 2 to 1000, only the 20 minutes to perform the 
model forecast is needed. 

\item Using {\F SCRIP} and {\F MPI}, running on 12 processors, the 
calculation of weights will not be done in parallel, will be slow, 
and will need to be computed each time. For all forecasts 1 to 1000, 
this may take 40 minutes to calculate remapping weights and 20 minutes 
to perform the model forecast.  

\end{list}

In some cases, SCRIP-WW3 will return suspicious values for some points, which
will result in warning message(s) in the screen output. When this occurs for a
small fraction of grid points, our experience (from analysis of the diagnostic
output in the {\file .nc} files) is that the remapping weights are valid,
since the problem points are at edges where \ws\ does not use the
weights. However, when this occurs for a large fraction of grid points, it is
likely that SCRIP-WW3 has truly failed. In this case, \ws\ stops with an error
message. Our experience is that this occurs most often for overlapping regular
grids with a large number of coincident line segments. A workaround exists: it 
can be remedied by adding an artificial offset to one of the grids. It was already 
possible to specify an offset in {\file ww3\_grid.inp} in the grid description,
but since that offset is intended as a real quantity, this other, artificial 
offset is implemented separately as a namelist option. It is {\code GSHIFT} 
under namelist group {\F MISC}. An example namelist would be: {\code MISC 
GSHIFT = 1.0D-6}. A smaller number will result in less regridding error, 
though the number must be sufficient large to actually have the intended 
beneficial effect. We recommend to determine this by trial-and-error, varying 
by factor 10 each time. 

\vssub
\subsection{~Limitations} \label{sec:scripE}
\vssub

Two features are not yet addressed, and will be addressed in a later version:
\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 15mm \labelwidth 7mm
             \rightmargin 5mm \itemsep 0mm \parsep 0mm}
\item  Communication between equal rank grids is still limited to regular
  grids. If one of the grids is irregular or unstructured, {\file ww3\_multi}
  will terminate with an error message. It is possible to have non-regular
  grids as part of a multi-grid system which includes equal rank grids, as
  long as the overlapping equal-ranked grids are all regular.

\item The ``input grid'' (or ``{F  modid}'') option for defining input fields
  (e.g. winds) is not implemented yet for irregular or unstructured grids. If
  this is attempted, {\file ww3\_multi} will terminate with an error
  message. The ``native'' input grid option should be used instead.

\end{list}

\noindent
\textrm{\textit{\underline{Attribution statement}: This section was written by
    E. Rogers. The coding and testing for this effort was performed by
    E. Rogers, M. Dutour, A. Roland, F. Ardhuin, and K. Lind. Technical advice
    was given by H. Tolman and T. Campbell.}}

%\bpage \pagestyle{empty}
